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Outline 


■  The  Generic  SE  Process  -  Concept  to  Initial  Operational  Capability 
(IOC) 


■  What  we  have  now 

■  Joint  Capability  Integration  and  Development  System  (JCIDS) 

■  DoDD5000  /  National  Security  Space  Policy  03-01 

■  What  AFIT  teaches 

■  What  lessons  do  the  case  studies  contain 

■  Observations 
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Visual  Model 


■  Systems  Engineers  often  visual  the  activities  and  task  to 
“pull  this  thread”  in  a  V-model. 


Time 


+► 


from:  Forsberg,  Mooz,  Cotterman;  “Visualizing  Project  Management” 
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Concepts  to  IOC 


Refinement  Development  &  Demonstration 

/V  Concept  „De?isn  _ 


,  IOC 

FOC 

Production  & 

Operations  & 

Deployment 

Support 

LRIP/IOT&E  A  Decision 
v  Review 

Pre-Systems  Acquisition 


Systems  Acquisition 


Acquisition 
(DOD  5000/NSS  03-01) 
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What  AFIT  teaches  in  Systems 

Engineering 


FAA 


Concept  Definition  and  Systems  Analysis 

-  Analysis  Guides,  Optimization,  Utility  Theory, 

-  Cost  Estimation,  Decision  Making/  Modeling 

-  FAA,  FNA,  FSA 


Additional  Emphasis  Needed 
To  Bridge  Seams 


FNA 


Decision  Analysis 


FSA 

Modeling  and  Simulation 


Systems  Architecture  (DoDAF) 

-  good  architecting  practices  6.  context,  evaluation 


Reliability  Analysis 
Increments 


Systems  Engine^ng  Design 
CD  -  requirement  j  specification, 

OA  trades/  allocation,  modeling 
ifica+  on/  test  planning 

M/S  B  Systems  Engineering  Management 

-  critical  path  analysis 

PDFl  -  EVM,  case  studies 

Software  Engineering 

^~CDR 


Risk  Analysis 
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Systems  Engineering  Case 

Studies* 


U5.AIRFORCE 


C-5  Galaxy 


Hubble  Space  Telescope 


Currently  available  case  studies 
provide  good  and  bad  examples 
of  pre-acquisition  activity 

-  New  cases  underway 


F-111 


JASSM  (FOUO) 


TBMCS  (Theater  Battle  *  Available  at: 

Management  Core  Systems) www.afit.edu/cse/ 
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Case  Study  Process 


a  Highly  Disciplined 

■  Depth  of  Study 

■  Number  of  Interviews 

■  Amount  of  Data  sources  and  access 

■  First  and  second  reviews  of  draft  case 

■  Objectivity  of  authors  and  editors 

■  Wide  review 


■  Scope 

■  Concentration 

■  Effort 

■  1-2MY/ 12-18  month 


This  is  Hard! 
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C-5  Galaxy 


■  Over  34  years  of  successful  operational  performance  in  support  of 
the  Nation’s  cargo/transport  needs 

■  USAF  inventory  of  126  C-5  aircraft  :74  C-5A,  50  C-5B,  2  C-5C 

■  During  Operation  Desert  Storm,  C-5  fleet  carried  46%  of  the  total 
inter-theater  cargo,  flying  only  29%  of  the  cargo  missions 

■  In  Operation  Iraqi  Freedom, 
the  C-5  fleet  carried  48%  of  total 
cargo  flying  only  23%  of  the  cargo 
missions 


13 


Mapping  the  C-5  to  Today’s  Process 


LF.5,A1R  FORCE 


•  Solid  early  Trades  studies 

•  Mock-ups  for  roll-on,  roll-off 

•  Stable  requirements 

•  Size,  Weight  trades,  Materials 

•  Mission  Analysis 


C-5  Galaxy 


•  Requirement  Management 
(Contract  award  &  beyond) 
-  Prioritization  of  Rqmts 


Sustainment 


Significant  structural 
consequences  (rewing) 
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C-5  Learning  Principles 


■  LP  #1 .  Systems  requirements  need  to  integrate  the  User  (warfighter), 
planners,  developers,  and  technologists  into  a  well-balanced,  well- 
understood  set  of  requirements 

■  LP  #2.  Total  Package  Procurement  Concept  (TPPC)  was  a  fixed-price, 
incentive  fee  contract  strategy  for  the  design,  development,  and 
production  of  58  aircraft.  Invented  to  control  cost  growth,  it  was  the 
underlying  cause  for  the  overrun 

■  LP  #3.  A  Weight  Empty  Guarantee  was  included  in  the  specification 
and  in  the  contract  as  a  cost  penalty  for  each  delivered  overweight 
aircraft.  This  measure  dominated  the  traditionally  balanced 
requirements  resulting  in  a  major  shortfalls  in  wing  and  pylon  fatigue 
life 

■  LP  #4,  Independent  Review  Teams  (IRTs)  were  to  assemble  national 
experts  to  examine  the  program  and  provide  the  best  advice  and 
recommendations  to  the  government  in  structures  design,  technology 
and  service  life 
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F- 111  System  Description 


■  In  1950s,  USAF  needed  a  replacement  for  F-1 00,  F-101,  and  F-105 
fighter-bombers 

■  Mach  2+,  60,000  foot  altitude 

■  All-weather  fighter,  originally  specified  as  capable  of  vertical  and 
short  takeoff  and  landing  (V/STOL) 


■  Many  firsts 

■  1st  terrain-following  radar,  allowing 
low  altitudes 

■  1st  production  aircraft  with  variable 
swing  wings 

■  1st  crew  escape  module 


it  to  fly  at  high  speeds  and 
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Mapping  the  F-111  to  Today’s  Process 


U5.AIRFORCE 


Air  Force 


•  Services  understood  and  conveyed 
Capabilities  and  Gaps 


*  Incompatible  and  Infeasible 
(Joint)  Requirements  -  “80%  commonality” 


F-111 


Sustainment 


•  Inflexible  requirements 
forced  unsatisfactory  design  trades 

•  Weight  growth  beyond  acceptable  range 


Navy  bows  out 

AF  “stuck”  with  Navy  design  features 


U.S.AIRFORCE 


F-111  Synopsis 


■  LP  #1 :  Ill-conceived,  difficult-to-achieve  joint  requirements  and  attendant 
specifications  made  the  F-111  system  development  extremely  costly,  risky  and 
difficult  to  manage. 

■  LP  #2:  Systems  Engineering  managers  (both  Gov’t  and  contractor)  were  not 
allowed  to  make  the  important  tradeoffs  that  needed  to  be  made  in  order  to 
achieve  an  F-111  design  that  was  balanced  for  performance,  cost  and  mission 
effectiveness  (including  survivability)  and  the  attendant  risk  and  schedule 
impacts. 

■  LP  #3:  The  F-111  suffered  from  poor  communications  between  the  Service 
technical  staffs,  and  from  over-management  by  the  Secretary  of  Defense  and  his 
staff,  which  restricted  the  System  Program  Office  (SPO)  Director  from  applying 
sound  systems  engineering  principles. 

■  LP  #4:  The  F-111,  like  any  complex  weapon  system  development  program 
which  provides  new  war-fighting  capability,  had  areas  of  risk  that  came  to  light 
during  RDT&E  even  though  there  was  perceived  low  risk  in  the  design. 

■  LP  #5:  Cancellation  of  the  Navy  F-111B  in  1968,  after  the  bi-service  design  was 
frozen,  and  production  of  the  Air  Force  F-111  A  was  well  underway,  had  a  lasting 
impact  on  the  United  States  Air  Force  F-111  performance  and  cost. 
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Hubble  System  Description 


Suit  HMliOr 


ekttlronfc  bw*cs 


inilftifrti-ht  module 

4ft*Jiroudl 

guidance  seflsore 


prtnuiy  mintir 


sofar  p.int-k 


radio  antenna 


wjl.ir  ji  jnrij 


aperturrdwjr 


Launched  in  1990,  scheduled 
operation  through  2010 
Permanent  space-based 
observatory  -  planned  regular 
servicing  missions 
2.4-meter  reflecting  telescope 
deployed  in  low-Earth  orbit 
(600  kilometers)  by  the  Space 
Shuttle  Discovery 
Complement  of  science 
instruments,  spectrographs 
cameras  and  fine  guidance 
sensors  operating  near-infrared 
into  ultraviolet  spectrums 
providing  resolution  of  0.1  arc- 
seconds 
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HST  Successful  System 


Over  100,000  observations  of  more  than  20,000  targets  have 
been  captured  for  retrieval 


/ 


Tadpole  Galaxy 
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,;f  Mapping  Hubble  to  Today’s  Process 

U.S.  AIR  FORCE 

Hubble  Space  Telescope 


•  Solid  early  Trades  studies 

•  Size/Cost  effectiveness  of  primary  optics 

•  Marriage  to  Shuttle  as  launch  vehicle 
>  On-orbit  services/  logistics  planning 


Created  “Institute”  to  advocate 
Requirements  for  the  world-wide 
Scientific  community 


•  No  end-to-end  full  integrated 
test  planned,  nor  executed 


Sustainment 


*  Poor  risk  mitigation 
Follow-through 


Poor  Quality  control 

Optics  grinding/polishing  setup  flawed 


U.S.AIRFORCE 


Hubble  Learning  Principles 


■  LP  #1.  Early  and  full  participation  by  the  customer/user  throughout  the 
program  is  essential  to  program  success. 

■  LP  #2.  The  use  of  pre-program  “Phased  Studies”  to  broadly  explore  technical 
concepts  and  alternatives  is  essential  and  provides  for  a  healthy  variety  of 
inputs  from  a  variety  of  contractors  and  government  (NASA)  centers. 

■  LP  #3.  Provision  for  a  high  degree  of  systems  integration  to  assemble,  test, 
deploy  and  operate  the  system  is  essential  to  success  and  must  be  identified 
as  a  fundamental  program  resource 

■  LP  #4.  Life  Cycle  Support  Planning  and  Execution  must  be  integral  to  design. 
Programs  structured  with  real  life  cycle  performance  as  a  design  driver  will  be 
capable  performing  in-service  better,  and  will  be  capable  of  dealing  with 
unplanned,  unforeseen  events  (even  usage  in  unanticipated  missions). 

■  LP  #5.  For  complex  programs,  the  number  of  players  (government  and 
contractor)  demands  that  the  program  be  structured  to  cope  with  high  risk 
factors  in  many  management  and  technical  areas  simultaneously. 
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B-2  Spirit 


■  Multi-role  bomber  combining  survivability  with  ability  to  deliver 
massive  firepower 

■  Second  generation  stealth  technology 

■  Unique  aerodynamic  control  scheme 

■  High  altitude  delivery  of  precision  guided  munitions 

■  Combat  proven 

■  33%  of  Serbian  targets  in 
opening  weeks  while  operating 
from  CONUS  base 

■  Wide  variety  of  strike  missions 
in  Afghanistan,  Iraq 
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Mapping  the  B-2  to  Today’s  Process 
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B-2  Learning  Principles 


■  LP  1,  Integration  of  the  Requirements  and  Design  Processes:  integration  of  the  SPO 
requirement’s  team  with  the  contractors’  design,  manufacturing  and  logistics  Work 
Breakdown  Structure  (WBS)  Task  teams  facilitated  continual  trade  studies  to  assess  the 
performance  trade-offs  against  schedule,  cost,  and  risk. 

■  LP  2,  WBS  Task  Teams  and  Functional  Hierarchy:  The  contract  Work  Breakdown 
Structure  (WBS)  stipulated  the  entire  program  content  and  tasking  and  the  company 
organized  the  design/development  effort  into  multiple  teams  according  to  the  WBS.  A 
vital  distinction  from  many  of  today’s  IPTs  was  retaining  the  WBS  Task  Team 
membership  throughout  the  functional  organizations’  various  management  levels. 

■  LP  3,  Air  Vehicle  Reconfiguration:  The  identification  of  a  major  aeronautical  control 
inadequacy  just  four  months  prior  to  the  formal  Configuration  Freeze  milestone 
necessitated  a  substantially  revised  design.  While  the  program  response  to  the  crisis 
was  rapid  and  effective,  the  magnitude  of  the  impact  on  the  downstream  cost  and 
schedule  was  not  anticipated  by  the  management  team  nor  predicted  by  the  systems 
engineering  process. 

■  LP  4,  Subsystem  Maturity:  The  effect  of  the  reconfiguration  on  the  maturity  of  all  the  air 
vehicle  subsystems  was  far  greater  than  projected.  It  took  longer  than  anticipated  by  the 
systems  engineering  process  to  recognize  the  growing  problem  of  getting  all  the 
specifications  updated.  These  iterations  after  PDR-2  resulted  in  the  Vehicle  Subsystems 
not  achieving  the  Critical  Design  Review  (CDR)  milestone  concurrently  with  the 
Structure,  but  rather  five  months  later. 

■  LP  5,  Risk  Planning  and  Management:  The  program  was  structured  so  that  all  risks 
affecting  the  viability  of  the  weapons  system  concept  were  identified  at  contract  award 
and  were  structured  as  part  of  the  program  WBS  work  plans.  Those  initial  risks  were 
closed  prior  to  PDR  2.  The  risk  closure  process  continued  throughout  development  and 
identified  new  risks  and  continuously  identified  new  risk  closure  plans. 
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TBMCS  System  Description 


a  Theater  Battle  Management  Core  System  (TBMCS)  is  an 
integrated  air  command  and  control  (C2)  system 

■  Performs  secure,  automated  air  battle  planning  and  execution 
management  for  Air  Force,  multi-service,  and  allied  commanders 

■  Provides  the  means  to  plan,  direct,  and  control  all  theater  air  ops 
and  to  coordinate  with  land, 
maritime,  and  special  ops  elements 

■  Modular  and  scalable  for 
air,  land,  or  sea  transport  and 
the  deployed  configurations  can 
be  tailored  to  meet  a  particular 
contingency 
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TBMCS  Successful  System 


■  Deployed  worldwide  as  the  mandated  joint  system  that  the  JFACC  uses  to  plan, 
manage,  and  execute  the  air  battle 

■  Demonstrated  very  rich  functionality:  it  can  produce  a  very  complicated  integrated  air 
battle  plan 

■  During  Operation  Iraqi  Freedom  (OIF),  the  size  of  the  Air  Tasking  Orders,  which 
planned  all  sorties,  well  exceeded  system  performance  parameters 


Total  Sorties  Flown 

41,404 

USAF 

24,196 

USMC 

4,948 

USN 

8,945 

USA 

269 

United  Kingdom 

2,481 

Australian 

565 

TBMCS  in  the  Air  Operations  Center,  Al-Udeid,  Qatar 
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TBMCS  Synopsis 


■  LP  #1 :  The  government  did  not  produce  a  Concept  of  Operations,  key 
operational  performance  parameters,  or  a  system  specification  for  the 
contractor 


■  LP  #2:  The  high-level  system  architecture  and  the  government’s  mandates 
for  software  reuse  and  use  of  commercial  software  (COTS)  products  were 
contradictory  and  problematic  for  the  system  development 


■  LP  #3:  The  system  and  subsystem  design  was  severely  hampered  by  the 
complexity  of  legacy  applications  and  misunderstanding  of  the  maturity  and 
complexity  of  commercial  and  third  party  software 


■  LP  #4:  Systems  and  interface  integration  was  highly  complex  -  integrating 
third  party  software  was  an  arduous  process  and  required  extensive 
oversight. 


■  LP  #5:  The  lack  of  a  firm  requirements  baseline  made  validation  and 
verification  very  difficult.  The  scheduled-driven  program  often  ran  parallel 
tests  without  clear  measures  of  success.  Not  being  able  to  replicate  the 
operational  environment  prior  to  acceptance  test  created  severe  problems. 


29 


LF.S.AIR  FORCE 


JASSM  System  Description 


■  Joint  Air-to-  Surface  Standoff  Missile  (JASSM)  is  an  autonomous, 
stealthy,  long  range  conventional,  air-to-ground,  precision  standoff 
missile  used  by  the  US  Air  Force  and  US  Navy 

■  Destroys  high  value,  well  defended  fixed  or  relocateable  targets, 
from  ranges  of  over  200  nm 

■  Employed  as  a  fully 
autonomous  "Fire  and  Forget" 

Weapon 

■  IOC  in  2003 
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Mapping  JASSM  to  Today’s  Process 


U*5.  AIR  FORCE 


Emerged  from  ashes  of  TSSAM 

-  Requirements  overload 

-  TSSAM  Cost  tripled 


Acq  Reform  Pilot 
-Only  2  performance  req’ts 
with  1  at  MOE  level 
-Contractor  responsible 
for  system  trades 


Some  stakeholders  unhappy 
with  resulting  system  trades 
-System  Reliability 
-Un-implemented  data  link  on 
base  JASSM 
-JASSM  production, 
JASSM-ER  program  on  hold 


Minimal  DT  program 

-  M&S  replaced  robust  flight  test  program 

-  Insufficient  data  to  address  reliability  concerns 
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JASSM  Synopsis 


■  LP  #1 :  JASSM  implemented  many  OSD  and  SAF/AQ  acquisition  reform 
initiatives  with  mixed  results 

■  Increased  Value  on  Past  Performance,  Mandated  No-Mil  Specs/Standards 

■  Implemented  Requirements  Control  Working  Group 

■  Applied  Performance  Based  Specification,  Configuration  Control  to  Contractor 

■  Used  Contractor-centric  Test  and  Evaluation  (T&E)  Plan 

■  Elevated  Importance  of  System  Affordability,  Rolling  Down-Select 

■  LP  #2:  APPLICATION  OF  CAIV  -  Use  of  many  COTS/NDI  components  and 
employment  of  non-traditional  processes  and  suppliers.  Objective  req’ts  traded 
off  for  lower  cost. 


■  LP  #3:  GOVERNMENT  TECHNICAL  OVERSIGHT  -  Less  than  directed  for 
traditional  MDAP  ACAT  1  programs,  especially  during  transition  to  production 
and  deployment  phases 

■  LP  #4:  INTERPRETATION  OF  TRADE  SPACE  -  Contractor  given  responsibility 
for  all  system  performance  below  range  and  Missile  Mission  Effectiveness 
(MME),  an  MOE  level  capability  (55  missiles  to  destroy  a  17  target  set).  The 
contractor  chose  to  design  JASSM  with  high  values  for  major  elements  of  MME, 
which  allowed  a  design  that  had  lower  free  flight  reliability. 

■  LP  #5:  USE  OF  MODELING  &  SIMULATION  -  Resulted  in  a  small  developmental 
flight  test  effort.  Insufficient  flight  tests  were  scheduled  to  adequately  address 
emerging  concerns  with  respect  to  missile  reliability. 
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Conclusions  from  Case  Study 

Assessment 


■  Systems  Engineering  exists  as  a  Continuum 

■  From  the  beginning  of  the  idea  ... 

■  ...  to  the  disposal  of  the  equipment. 

■  There  are  no  shortcuts 

■  Different  tools,  people,  skills  are  necessary 
throughout  the  modified  “V” 

■  Teach  the  skills 

■  Train  and  retain  the  people  (ALL  associated  with 
implementing  the  SE  process) 
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Our  Management  Structure 


■  Development  Planning  (Capability  Planning) 

■  Existed  in  the  past 

■  Reported  to  Product  Center  commanders 

■  Funded  by  PE  65808 

■  Mission  focused 

■  Mission  narrowly  defined  in  today’s  view 

■  Not  adequate  for  Systems  of  Systems 
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Capability  Planning 


Management  Structure 


Development  Planning  (Capability  Planning) 

AS  A 

Technology 

_ 

~\  \  -\ 

FNA  FNA 

\ 

-  \ 
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Observations 


■  We  need  the  SE  process  to  function  end-to-end 

■  Transition  points  are  highest  risk  for  failure  -  “SE  escapes” 

■  Concept  Decision  -  AoA 

■  CDD  -  M/S  B 

■  Development  Planning  (Capability  Planning)  function  is  vital 

■  Must  be  capability  driven  -  analysis  must  span  multiple  domains 

■  Users  have  the  responsibilities,  but  neither  the  time  nor  the  skills 

■  Reconstituted  product  center  XR  shops 

■  Skills  not  yet  fully  developed 

■  Still  excessively  domain  specific 

■  Inconsistently  funded 

■  No  clear  role 

■  Technology  is  short-term  focused 


36 


18 


BACKUP 
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Case  Studies 
http://www.  afit.  edu/cse 


What  we  have  now  -  Assessment 


■  Usually  good  job  of  relating  JCIDS  effort  to  National 
Strategies,  QDR,  Joint  Concepts,  UJTL,  Air  Force  Capabilities 

■  But,  Task  Analysis,  per  JCIDS  and  good  Systems  Engineering 
practice,  should  define  the  standard  to  achieve 


Measures  of  Effectiveness  (MOE)  by  which  to  compare 
different  solutions 


Our  experience  supporting  AoAs  has  taught 
us  that  developing  a  good  set  of  MOEs  is 
usually  a  harrowing  business. 

-  Air  Force  Analyst’s  Handbook 


LF.S.AIR  FORCE 


What  we  have  now  -  Assessment 


■  Without  good  Measures  of  Effectiveness,  any  Gaps  in  the 
Capability  Assessment  (Functional  Needs  Analysis)  would  be 
“hand  waiving” 

■  Results  in  not  giving  a  sound  and  full  understanding  of  gaps / 
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❖ 

U-5,  AIR  FORCE 


JCIDS  Analysis 


Area 
Analysis 

Functional 

Solutions 

Analysis 


OPLANS 

Defense 

and 

H 

Planning 

Functional 

CONPLANS 

T — 7 

Scenarios 

Functional 

Needs 

Analysis 


Reconciliation 
&  Recommendations 


Decision 

and 

Action 


Science  & 
Technology 


DOTMLPF  Changes 


Experimentation 


Adapted  from  CJCSI  31 70.01  C  Joint  Capabilities  and  Integration  Development  System,  Figure  A-1 


DAU  Acquisition  Guide  2006 
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JCIDS  Analysis 


U»5,A!R  FORCE 


■  Functional  Area  Analysis  (FAA) 

■  Identify  operational  tasks,  conditions,  and  standards  needed  to 
accomplish  military  objectives 

■  Result:  Tasks  to  be  reviewed  in  the  FN A 

■  Functional  Needs  Analysis  (FNA) 

■  Assess  ability  of  current  and  programmed  capabilities  to 
accomplish  the  tasks 

■  Result:  List  of  capability  gaps 

■  Functional  Solutions  Analysis  (FSA) 

■  Operational  based  assessment  of  doctrine,  organization,  training, 
materiel,  leadership/education,  personnel,  and  facilities 
(DOTMLPF)  approaches  to  solving  capability  gaps 

■  Result:  Potential  integrated  DOTMLPF  approaches  to  capability 


gaps 


■  Post  Independent  Analysis 

■  Independent  analysis  of  approaches  to  determine  best  fit 

■  Result:  Initial  Capabilities  Document 
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